EBM Workshop

Adam La Caze

Overview

Objectives

  • Critically reflect on the what, why and how of evidence-based medicine
  • Develop and practice your skills in critical appraisal and clinical epidemiology
  • Be able to confidently interpret and apply clinical research in your practice

What is EBM?

My short version…

EBM is a commitment to using best evidence to inform therapeutic decisions

Most quoted

Evidence based medicine is the conscientious, explicit, and judicious use of current best evidence in making decisions about the care of individual patients. The practice of evidence based medicine means integrating individual clinical expertise with the best available external clinical evidence from systematic research.

Sackett et al. (1996)

Greenhalgh and Donald

EBM is the use of mathematical estimates of the chance of benefit and the risk of harm, derived from high-quality research on population samples, to inform clinical decision-making

Greenhalgh (2012)

There are many components to EBM

  • Individual therapeutic decision-making
  • Guidance on use of evidence: hierarchies of evidence; GRADE
  • Critical appraisal
  • Systematic review and meta-analysis methods
  • Guideline development
  • Systematic summaries of evidence
  • Meta-research methods
  • Clinician and patient decision aids
  • Clinical epidemiology

Levels of evidence (OCEBM Levels of Evidence Working Group 2011)

GRADE

GRADE: Grading of Recommendations, Assessment, Development and Evaluation

GRADE offers a system for rating quality of evidence in systematic reviews and guidelines and grading strength of evidence of recommendations in guidelines

G. Guyatt et al. (2011) p. 384

GRADE Strength of Evidence

GRADE Strength of Recommendation

Factors in considering strength of recommendation

  • Uncertainty about balance between desirable and undesirable effects
  • Uncertainty or variability in values in preferences
  • Uncertainty about whether the intervention represents a wise use of resources

G. H. Guyatt et al. (2008)

Why EBM?

Flecainide

  • Certain types of ventricular arrhythmia are a risk factor for death following myocardial infarction
  • Prior to 1991, it was common to prescribe antiarrhythmic agents to post-MI patients with asymptomatic or mildly symptomatic ventricular arrhythmias (Morganroth, Bigger, and Anderson 1989)

CAST Trial (CAST Investigators 1991)

PICO CAST
Participants Patients 6 days to 2 years following myocardial infarction with ventricular premature depolarizations who responded to antiarrythmic agent.
Intervention Flecainide, encainide, moricizine
Comparator Placebo
Outcome Death or cardiac arrest with resuscitation due to arrhythmia

CAST: Outcome

  • Primary endpoint death or cardiac arrest due to arrhythmia: 5.7% v 2.2% intervention v control (p = 0.0004)
  • Total death or cardiac arrest: 8.3% v 3.5% intervention v control (p = 0.0001)

Primary endpoint for CAST trial

Hormone Replacement Therapy

  • From 1980–2000 many women were prescribed long-term hormone replacement therapy to reduce cardiovascular disease
  • This practice was backed-up by what what was understood about the mechanisms and several large observational studies (Stampfer and Colditz 1991)

Women’s Health Initiative (Women’s Health Initiative Investigators 2003)

PICO WHI
Participants Women 50–79 years who were postmenopausal.
Intervention Oestrogen plus progesterone (for women with an intact uterus) or oestrogen alone
Comparator Placebo
Outcome Coronary heart disease (acute myocardial infarction, death due to coronary heart disease, or silent myocardial infarction)

Women’s Health Initiative: Outcomes

  • The Women’s Health Initiative study (2003) randomized 16,608 postmenopausal women to HRT or placebo and followed them for 5 years
  • Overall hazard ratio for coronary heart disease 1.24 (95% unadjusted CI: 1.00–1.54)
  • Women randomized to HRT suffered an additional 6 cases of coronary heart disease per 10,000 person years

How do we explain the very different results of the observational studies compared to WHI?

Study design

Randomized trial

Cohort study

Confounder

Necessary conditions for a factor to be a confounding factor between exposure and outcome:

  1. The factor must be an extraneous risk factor for the outcome
  2. The factor must be associated with the exposure under study in the source population
  3. The factor cannot be an intermediate step in the causal path between the exposure and the outcome

Life-course socioeconomic status and HRT

How to practice EBM?

Key tools

  1. Pre-appraised resources & systematic summaries
  2. PICO
  3. Critical appraisal & clinical epidemiology

Pre-appraised resources

PICO

PICO Clinical Question Critical Appraisal
Participants What are the key characteristics of the patient(s) Who was recruited to the study? What were the inclusion/exclusion criteria? Who participated?
Intervention What is the intervention under consideration What intervention did the treatment group receive?
Comparator What is the comparator, control, or usual alternative What did the control or placebo group receive?
Outcome What is the patient-relevant outcome? (or society-relevant outcome?) What was the primary outcome of the trial?

Inference

Hypothesis testing

Underpowered tests

Overpowered tests

Interpreting results of tests with different power

Statistically significant result Non-statistically significant result
Adequately powered test Reject the null. Accept the alternative hypothesis The test failed to reject the null. Either the null is true or the effect size is smaller than was tested
Underpowered test Provisionally accept the alternative hypothesis Underdetermined result. The test is unable to detect effect sizes that might be important.

Confidence intervals

95% Confidence interval

Ways to think about a confidence interval…

Precise, but confusing

  • If the study was repeated many times, and the same procedure was used to calculate the 95% confidence interval, in the long run, you would expect the calculated 95% confidence intervals would include the true value of the parameter 95% of the time

  • A 95% confidence interval provides the range of values that are not statistically different from the observed point estimate at the 0.05 level

Less precise, but useful

  • The confidence interval provides a range of plausible values for the unknown parameter

  • The lower limit is a likely lower bound estimate of the parameter; the upper limit a likely upper bound

Incorrect and misleading

  • You can be 95% confident that the true value lies between the observed confidence interval

  • The 95% confidence interval has a 95% chance of including the true effect size

Intention to treat

Coin flipping

Random error

What are your expectations when you toss a fair coin 10 times?

##  [1] 0 0 0 1 1 0 1 1 0 1
##  [1] 1 0 0 1 1 1 1 1 0 0
##  [1] 0 1 1 0 1 1 0 1 1 1

In these series of experiments we saw 5, 6 and 7 \(H\).

##  [1] 5 8 5 5 7 5 7 4 7 4

  • Some 10 coin toss experiments provided 0 \(H\) or 10 \(H\)
  • Most provided an outcome close to 5 \(H\)
  • The mean number of \(H\) across all of these experiments was 4.998

Take home

  1. Random error can make it hard to make inference from small samples of data.
  2. Increasing the number of repetitions provided mean results much closer to what we expect given the coin is fair.

Task

Imagine you have coin of unknown bias (i.e. the probability of \(H\) is unknown—it is unknown whether the coin is fair, favours \(T\) or favours \(H\)). What test could you conduct to assess whether the coin is fair?

Attempt to describe the hypothesis you are testing and the statistical model you are using for the test.

Coin of unknown bias

The mean number of \(H\) for the fair coin was 4.998 and for the coin of unknown bias was 6.9982.

CAST Investigators. 1991. Mortality and morbidity in patients receiving ecainide, flecainide or placebo.” New England Journal of Medicine 324 (12): 781–88.
Greenhalgh, T. 2012. Why do we always end up here? Evidence-based medicine’s conceptual cul-de-sacs and some off-road alternative routes.” Journal of Primary Health Care 4 (2): 92–97. https://www.rnzcgp.org.nz/assets/documents/Publications/JPHC/June-2012/JPHC-Vol-4-No-2-June-2012.pdf{\#}page=5.
Guyatt, G H, A D Oxman, R Kunz, Y Falck-Ytter, G E Vist, A Liberati, and H J Schunemann. 2008. Going from evidence to recommendations.” BMJ 336 (7652): 1049–51. https://doi.org/10.1136/bmj.39493.646875.AE.
Guyatt, Gordon, Andrew D Oxman, Elie A Akl, Regina Kunz, Gunn Vist, Jan Brozek, Susan Norris, et al. 2011. GRADE guidelines: 1. Introduction.” Journal of Clinical Epidemiology 64 (4): 383–94. https://doi.org/10.1016/j.jclinepi.2010.04.026.
Holman, Bennett, and Justin P Bruner. 2017. Experimentation by industrial selection.” Philosophy of Science 84 (5): 1008–19.
Morganroth, Joel, J Thomas Bigger, and Jeffrey L Anderson. 1989. Treatment of ventricular arrhythmias by United States cardiologists: a survey before the Cardiac Arrhythmia Suppression Trial Results were available.” American Journal of Cardiology 65: 40–48.
OCEBM Levels of Evidence Working Group. 2011. The Oxford 2011 Levels of Evidence.” Technical report. Oxford Centre for Evidence-Based Medicine. http://www.cebm.net/ocebm-levels-of-evidence/.
Sackett, David L, W. M C Rosenberg, J A Muir Gray, R B. Haynes, and W Scott Richardson. 1996. Evidence based medicine: what it is and what it isn’t.” BMJ 312 (7023): 71–72. https://doi.org/10.1136/bmj.312.7023.71.
Stampfer, M J, and G A Colditz. 1991. Estrogen replacement therapy and coronary heart disease: a quantitative assessment of the epidemiologic evidence. Preventive Medicine 20 (1): 47–63.
Women’s Health Initiative Investigators. 2003. Estrogen plus progestin and the risk of coronary heart disease.” New England Journal of Medicine 349 (6): 523–34. https://doi.org/10.1056/NEJMoa1404304.